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years, CRESST has been developing and evaluating a series of software 
products to measure various types of learning. Collectively, these are called 
the Integrated Assessment System. The integrated simulation CRESST has 
developed includes both collaborative and individual concept mapping tasks, a 
problem-solving search task, an explanation task, and a metacognitive 
questionnaire. New tools should measure one or more aspects of the CRESST 
model of learning, be stand-alone or integrate with other tools, have 
authoring capacity, use automated scoring, and be Internet -deployable . 
Specifications are listed for eight proposed tools, and each is evaluated for 
the CRESST requirements. The tools are: (1) Collaborative Concept Mapper, a 

tool for group use; (2) Flowcharter, a tool to allow students to sequence 
events; (3) Idea Generator and Evaluator, a tool for group problem solving; 

(4) Model Simulator, a model building tool; (5) Networked Team Simulator, a 
team model builder; (6) Multimedia Concept Mapper, a multimedia tool for 
concept mapping; (7) Outliner, an organizing tool; and (8) Problem Solver, an 
instrument to measure problem solving processes. (Contains 4 tables and 13 
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ESTIMATE OF THE POTENTIAL COSTS AND EFFECTIVENESS OF SCALING 
UP CRESST ASSESSMENT SOFTWARE 1 

Gregory K. W. K. Chung 2 , Howard E. Herl 2 , 

Davina C. D. Klein 2 , Harold F. O'Neil, Jr. 3 , and John Schacter 2 

In this report, we examine issues in the scale-up of CRESST assessment 
software. We use the term "scaling up" in a metaphorical sense, meaning what 
new assessment tools should be added to our assessment software. Criteria for 
new assessment tools are based upon the need to develop additional evaluation 
tools that would be useful and relevant to students, teachers and the business 
community. 

During the past several years CRESST has been developing and evaluating a 
series of software products to measure various types of learning. These are 
collectively called the Integrated Assessment System. This system has been 
documented in a series of reports to the Department of Defense's Computer 
Aided Education & Training Initiative (CAETI) (Baker & O'Neil, 1996; Herl & 
O'Neil, 1996; Herl et al., 1996; Klein, O'Neil, & Baker, 1996) and a recent series of 
conference papers presented at the 1997 annual meeting of the American 
Educational Research Association (Chung, O'Neil, Herl, & Dennis, 1997; Herl, 
O'Neil, Chung, & Dennis, 1997; Klein, O'Neil, Dennis, & Baker, 1997; O'Neil, 
1997; Schacter et al., 1997). These products are model-based in the sense of 
reflecting the five types of learning that are specified in the CRESST model of 
learning (Baker, 1995; see Figure 1). 

CRESST has assembled a suite of performance assessment tasks (our 
integrated simulation) onto which we have mapped the types of learning 
expected of students. The design of integrated simulation performance 
assessment has the following characteristics: (a) relevant, project-based scenarios 
that include meaningful, real-world tasks; (b) individual and team processes and 
products; (c) a technology base using Web-based, networked systems; and (d) 
model-based assessments that integrate types of cognitive learning, grade level, 
and content area. 



1 Authors are listed alphabetically. 

2 CRESST/University of California, Los Angeles 

3 CRESST/University of Southern California 




Figure 1. The CRESST model of learning. 



The integrated simulation we have developed includes both collaborative 
and individual concept mapping tasks, a problem-solving search task, an 
explanation task, and a metacognition questionnaire. It is designed to be useful to 
students at different ages, given different levels of understanding, and with 
different content areas. Its administration is computer-based; the software 
controls and documents progress, while permitting real-time scoring and 
reporting. For assessing content understanding we use a concept mapping 
approach that is computer-based. To assess collaborative skills, we use a 
collaborative mapper (Chung et al., 1997), and to assess problem solving, we use a 
Web-based approach to support a problem-solving search task (Schacter et al., 
1997). To assess metacognition, we use a questionnaire to measure both trait and 
state metacognition (e.g., see O'Neil & Abedi, 1996, for a description of the state 
version). In general, the evaluation data for our assessment tools show them to 
be feasible. However, there is limited reliability and validity information. Such 
studies are planned. Moreover, there is an issue of scaling up such an integrated 
simulation: What new assessment tools should be added to the integrated 
simulation? 



New Computer-Based Assessment Tools 

The proposed assessment tools will support Project 1.3 goals of the CRESST 
grant proposal to the Office of Educational Research and Improvement (Baker, 
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Linn, & Herman, 1995), particularly the goals for Advanced Technology and 
Quality Improvement and Utility Enhancement. We suggest that basic 
assessment research of such tools through prototype development would be 
funded by Project 1.3, and product refinement and possible commercialization 
would be funded through other sources (e.g., NCES or commercial partners). 

In general, two classes of software are needed to design and develop 
computer-based automated assessment tools: (a) individual assessment tool 
applications software, and (b) application programming interfaces (APIs). 
Application programming interfaces provide transparent layers for collecting, 
parsing, managing, analyzing, and outputting data. Application programming 
interfaces provide assistance for tool evaluation and performance (e.g., data 
logging and front-end and back-end interface configuration). End users (e.g., 
students and trainees) interact directly with the application software assessment 
tools (for example, the Multimedia Concept Mapper) but indirectly with the 
application programming interfaces. 

Tool Design Strategy 

Our strategy is to design assessment tools that (a) measure one or more 
components of the CRESST model of learning, (b) are stand-alone or integrate 
with other tools, (c) have authoring capability, (d) employ automated scoring and 
reporting, and (e) are Internet-deployable in both Windows and Macintosh 
environments. 

Tool integration. Each tool will be designed as both a stand-alone 
instrument and part of an integrated assessment suite. To illustrate this point, 
the software package Microsoft Office is analogous to a tool suite or our 
integrated simulation, and the applications within Microsoft Office, such as 
Word, PowerPoint, and Excel, are analogous to individual component tools. To 
maximize integration, all tools will be designed around a set of application 
programming interfaces. These application programming interfaces are sets of 
code and services that are independent of any specific tool. All tools will collect 
and log users' performance and process data. Instead of creating (and recreating) 
code to perform data logging on a per-tool basis, the data logging application 
programming interface will provide one set of standard data logging routines 
accessible by all tools. Authoring, scoring, reporting, and analysis are other areas 
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that would benefit from the development of application programming interface 
software. 

Authoring. All tools will be authorable. That is, all tools will provide users 
with the capability to define content, establish scoring criteria, and customize the 
interface of the tool to adapt to audiences of different ages. For younger users, 
interfaces would be adjusted to be sparse and iconic, whereas for older, more 
experienced users, interfaces can be adjusted to show the more complex features 
of the software. An example of an adjustable interface is the Microsoft Word 
Tool Bar. Users can customize this tool bar such that either very few or many 
options appear on it. Our authoring capabilities will take a similar approach. 

Automated scoring and reporting. A goal of all CRESST applications is real- 
time automated scoring and reporting. Automated scoring and reporting 
processes involve real-time data collection, management, and analysis. 
Therefore, underlying application programming interfaces will be written to 
handle these real-time processes for all tools. Although each tool will have 
specific reporting requirements, many of the underlying functions (e.g., 
calculating statistics) will be handled by an application programming interface. 

Internet-deployable and platform-independent. All tools will be designed to 
be deployable across the Internet for both the Windows and Macintosh 
environments. Thus, tools will be developed in an architecture-neutral language 
such as Java™ or in applications that create downloadable plug-ins that run on 
both platforms. In addition, all tools will be compatible with Netscape 
Navigator™ and Microsoft Explorer™ Internet browsers. 

Spiral model of software development. Our assumption is that we will be 
developing software. Given this — and because such software will likely have 
evolving requirements — we believe a spiral model is the appropriate 
development model. The spiral model minimizes risks through incremental 
development of software. In concrete terms, periodic deliveries of prototypes are 
planned. The goal of each prototype cycle is to design and implement only the 
highest priority features. Each successive prototype cycle incorporates more 
features. The benefit of the spiral model is that technical and functional 
problems are identified early in the development cycle. This early risk 
identification means that we can make informed decisions about whether to 
cancel or continue a project. An added benefit of this development model is that 
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there is high product visibility: There is always a working prototype that can be 
demonstrated. 

Proposal assessment tools. Table 1 lists each proposal assessment tool and 
the component of the CRESST model that serves as the measurement target. 
Table 2 presents a list of assessment tools and their respective application 
programming interfaces. Detailed descriptions of each tool are found in the Tool 
Specifications section. 

Tool Specifications 

In this section, specifications for a series of tools are proposed. Table 3 
provides a first cut of the specifications for each proposed tool, and Table 4 
provides a first cut of specifications for each proposed application programming 
interface. 

Next Steps 

We need to prioritize the development activities. 
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Table 3 

Specifications for Assessment Tools 
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Development cycle 5-10 months production work and testing 

Large-scale assessment outlook Completely scalable 

(scalability) 



Table 3 (continued) 
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Large-scale assessment outlook Completely scalable 

(scalability) 



Table 3 (continued) 
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Development cycle 4-10 months 

Large-scale assessment outlook Completely scalable 

(scalability) 



Assessment Tool: Model Simulator 
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Assessment Tool: Networked Team Simulator 
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Table 3 (continued) 
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Table 4 

Specifications for Assessment Application Programming Interfaces (APIs) 



£ 



X 

03 

a. 

03 

U 

01 

3 

3 

X 



X 

01 



03 

01 

Im 

0 

01 

X 

3 

.5 

CO 

01 

U 

’> 

Ut 

01 

CO 

bo 

.5 

Ut 

o 

X 

3 

< 



a3 

ii 

O 03 



03 

CL 



CO 

01 

0 

> 

In 

01 

CO - 

bC 

.5 

Ui . 

O 

■s . 

3 

03 

a> 

x 

> * 
o 



CL X 

tj 



O 

o 



E 

< 



C 

a; 

s 



CO 
to 

a> 

CO CO 
CO 

X 03 



bO 

.5 



<u 
co 

O ~ 
Cl "H 
t- cu 

r? CL 
CL to 



CL 

< 

bo 

C 

X 

o 

X 

3 

< 



co 

5 

C 

o 

E 

(N 



cj 

U 

c 

01 

£ 

CL 

0 

*3 

> 

01 

D 



bo 

o 

C 

o 

E 

E 

o 

u 



01 

CO 

03 

X 

2 

3 

X 

03 

0 

X 

01 
be 
bo 

oj 

X 



03 

13 

X 



03 

X 



n 

£ 2 

to rc 

§ &■ 

1 s 



.5 

bO bO 
bO O 

0 — 

— o 

13 X 
t 3 <y 

01 3 

X > 



CL 01 
_ X 



< 8 

•g > 

[5 S 

cn 

ai 

to 4J 
O 03 
CL c 
C 

3 i “ l 



bC 

.5 

bC 

bC 

_o 

03 _ 

13 cl 

a < 



X 

c 

o 

E 

vp 

(N 



^ £ X 

K-l > u 



U 

u 

c 

01 

E 

Cl 

O 

*3 

> 

O) 

O 



03 

X 

01 

u 



E 

0 

M-l 

u 

01 
CL 
be 

.5 

> 

.01 

X 

3 

Ui 

•§ j 2 

-2 3 

y to 

.s a 

S g> 

.2 .S 

£ o 

01 Li 
CO ^ 

be 5P 

•5 -S 

0 o 

CL CL 

01 01 

i § 

£ m 

0 3 
** x 

co 

01 x 
tJ 01 

'E I 

a £ 

bO £ 
.5 oi 

§* o 

*“• CO 

01 u 

X X 

•— CO 

> X 

O 03 

V-t n-» 

CL ^ 
*£ 



^2 E- 
< 01 
CO ^ 

x 3 " 

H 03 

* • X 

01 

co w 
O co 
X 01 

CL cj 

t? O 

CL CL 



CL 

< 

bO 

.s 



0 

CL 

01 
X 



CO 

5 

c 

o 

E 

CN 



u 

>> 

u 

c 

01 

E 

CL 

js 

3 

> 

o> 

D 



3 
c r 
01 



03 

X 



X 

01 

u 

3 

cr 

01 



01 

to 

X 

Vh 

O 



2 s 

. o 



3 


u 


0 


E 




£ 


0 






X 


CO 


01 


01 


CO 


u 


o3 


’> 


X 

1 


Uh 


X 


01 


V- 


CO 


O 


01 


5 


> 




03 


3 

c 


Vh 

O 


Ut 

01 


X 

03 


£ 




0 


3 


Ui 


u 


O 



01 V 

3 -s 

> o 

2 5 

On E 

— 1 03 

X 01 

£ 3 

C 

CL .2 
< 3 

CO *2 

H jj 

s ° 

O 01 
CL g 

L ^ 

3 C 
On O 



01 

X < 

03 X 

01 

0 ^ 

11 

O ni 

U c 



5 

c 

o 

E 

00 

vi 



U 

c 

01 

E 

CL 

jD 

3 

> 

0 > 

D 



be 

.s 

x 

2 

03 

01 

CO 



0 

>v 

01 
X 



01 

.5 

bO 

S co 
- bO 

5 .5 

% x 
01 S 

CO g 

Oi X' 
X X 

3 3 
"3 3 
.5 6 

to w 
01 ^ 
.2 o 

6 X 

S X 



X X 
u ^ 
•2 O 
03 
01 
CD 

01 
. X 






0 03 

£ x 

£ £ 
t-i 

CO 

01 X 
u 01 



01 

CO 

X 

u 



03 

01 3 

CO C 

01 

3 * 

0 £ 

o .2 
— . 01 
X CL 

1 s 

11 

X CQ 



01 X 

to 

O ^ 

cl3 

-p 03 
Qu u 



CL 

< 

X 

cj 

Ui 

03 

01 

CD 



X 

c 

o 

E 

00 

t 



cj 

u 

c 

01 

E 

CL 

3 

> 

01 

O 



01 x 

■§ § 
C to 

bO 8 

.5 S' 

> o 
01 ^ 
X 01 
X bC 

2 a 

T3 5 

§ £ 

bO V5 

.5 p 

’*-> o 

a * 

<u “> 

x .5 

£ 3 

3 *> 

S 0 

to CL 
01 

u X 

£ S 



c .2 
.2 3 
3 E 
3 S 

.§s 

in 01 
to S 



co C 

01 01 

2 c 
> 

Uh 
01 



o 

CL 

E 

o 

u 

.2 so 

5 -5 
3 £? 
C 01 

.5 D 

co ^ 

01 

X 01 
’> ^ 
o g 

f-S 

> bO 

E c 

< 5 

to p 



01 
to 01 



03 



° E 

CL S 

p p 

3 03 X 

CL CL X 



CL 

< 

C 

o 

"•M 

£ 

3 

E 

Id 



CO 

C\J 



01 to 
01 



CO 

£ 

c 

0 

E 

00 

1 

00 



u 

u 

C 

01 

E 

CL 

3 

> 

01 

Q 



CO 

CM 



er|c 



15 



Text classifier API Purpose: This API will provide text classification services to tools. Given a set of text, this API will provide an estimate 
of similarity between the given text and other text in the database. This API will most likely be used for matching 
student written texts to prescored texts and categorizing short answers or user-typed messages. 
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self-regulation. The task must provide opportunities for these kinds of processes to occur. A strong coupling is required 
between overt action and these processes. 
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